Add Hashtable and LongHashingUtils utilities#11409
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a534e4f4f4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
|
|
||
| public static final int hash(Object obj0, Object obj1, Object obj2, Object obj3, Object obj4) { | ||
| return hash(hashCode(obj0), hashCode(obj1), hashCode(obj2), hashCode(obj3)); | ||
| return hash(hashCode(obj0), hashCode(obj1), hashCode(obj2), hashCode(obj3), hashCode(obj4)); |
- Add Support.insertHeadEntry(buckets, long keyHash, entry) overload that derives the bucket index itself. Callers that already have a hash but not the index (the common case) now avoid the redundant bucketIndex(...) hop. - D1.insert, D1.insertOrReplace, D2.insert, D2.insertOrReplace: use the new overload, drop the (thisBuckets local, bucketIndex compute, setNext, store) sequence at each call site. - D2.buckets: drop the `private` modifier to match D1.buckets. Both are package-private so iterator tests in the same package can drive Support.bucketIterator against the table's bucket array. Added a short comment on both fields documenting the rationale. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three follow-ups from the design review: - Make Hashtable.Entry.next private. All same-package readers (BucketIterator) already had a next() accessor; the leftover direct field reads now route through it. Closes the "mixed encapsulation" gap where some readers used the accessor and same-package ones reached for the field. - BucketIterator and MutatingBucketIterator now document that chain-walk work happens in next() (and the constructor for the first match); hasNext() is an O(1) field read. - Add D1.getOrCreate(K, Function) and D2.getOrCreate(K1, K2, BiFunction). Both reuse the lookup hash for the insert on miss, avoiding the double-hash that "get; if null then insert" callers would otherwise pay. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses PR #11409 review comments: - #3267164119 / #3267165525: wrap every single-line if/break body in braces (7 sites across BucketIterator, MutatingBucketIterator, and the full-table Iterator). - #3275947761 / #3275948108 (sarahchen6): null out the removed/replaced entry's next pointer after splicing it out of the chain in MutatingBucketIterator.remove / .replace. Applied the same fix to the full-table Iterator.remove for consistency. Rationale: detaching prevents accidental traversal through a removed entry via a stale reference and lets the GC reclaim a chain tail that the removed entry was the last referrer to. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
66ec7f6 to
e2642cd
Compare
…sistency Addresses PR #11409 review comment #3276167001. The method parallels the primitive hash(boolean) / hash(int) / hash(long) / ... family, so naming it hash(Object) -- with null collapsing to Long.MIN_VALUE as a sentinel distinct from any real hashCode -- matches the rest of the public surface. Test call sites that pass a literal null now disambiguate against hash(int[]) / hash(Object[]) / hash(Iterable) via an (Object) cast. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
| * <ul> | ||
| * Use cases include... |
There was a problem hiding this comment.
nitpick:
| * <ul> | |
| * Use cases include... | |
| * Use cases include... | |
| * <ul> |
Trim per-span work on metrics aggregator publish path
ConflatingMetricsAggregator.publish does a handful of redundant operations on
every span. None individually is large; together they show as ~2.5% on the
existing JMH benchmark once the benchmark actually exercises span.kind.
- dedup span.isTopLevel(): publish() reads it into a local, then shouldComputeMetric
read it again. Pass the cached value in.
- resolve spanKind to String once: master called toString() twice per span (once
inside spanKindEligible, once at the getPeerTags call site) and used HashSet
contains on a CharSequence (which routes through equals on String). Normalize
to String up front and reuse.
- lazy-allocate the peer-tag list: getPeerTags() always allocated an ArrayList
sized to features.peerTags() even when the span had none of those tags set.
Defer allocation until the first match; return Collections.emptyList() when
none hit. MetricKey already treats null/empty peerTags as emptyList, so no
behavior change.
Drop the spanKindEligible helper — the HashSet.contains call inlines fine in
shouldComputeMetric.
Update the JMH benchmark to set span.kind=client on every span. Without it the
filter path short-circuits before the peer-tag and toString work, so the wins
above aren't measurable. With it:
baseline 6.755 us/op (CI [6.560, 6.950], stdev 0.129)
optimized 6.585 us/op (CI [6.536, 6.634], stdev 0.033)
2 forks x 5 iterations x 15s. ~2.5% mean improvement and much tighter variance
fork-to-fork.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add SpanKindFilter and CoreSpan.isKind for bitmask-based kind checks
Introduce SpanKindFilter -- a tiny builder-built immutable filter whose state
is an int bitmask indexed by the span.kind ordinals already cached on
DDSpanContext. Each include* on the builder sets one bit (1 << ordinal); the
runtime check is a single AND against (1 << span's ordinal).
CoreSpan.isKind(SpanKindFilter) is the new entry point. DDSpan overrides it
to do the bit-test directly against the cached ordinal -- no virtual call,
no tag-map lookup. The two existing test-only CoreSpan impls (SimpleSpan
and TraceGenerator.PojoSpan, the latter in two source sets) implement isKind
by reading the span.kind tag and delegating to SpanKindFilter.matches(String),
which converts via DDSpanContext.spanKindOrdinalOf and does the same AND.
Refactor: DDSpanContext.setSpanKindOrdinal(String) now delegates to a new
package-private static spanKindOrdinalOf(String) so the same string-to-ordinal
mapping serves both the tag interceptor path and SpanKindFilter.matches.
This is groundwork -- nothing in the codebase calls isKind yet. The next
commit will replace the HashSet-based eligibility checks in
ConflatingMetricsAggregator with SpanKindFilter instances.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Use SpanKindFilter in ConflatingMetricsAggregator
Replace the two ELIGIBLE_SPAN_KINDS_FOR_* HashSet<String> constants and the
SPAN_KIND_INTERNAL.equals check with three SpanKindFilter instances:
METRICS_ELIGIBLE_KINDS, PEER_AGGREGATION_KINDS, INTERNAL_KIND. Eligibility
checks now go through span.isKind(filter), which on DDSpan is a volatile
byte read against the already-cached span.kind ordinal plus a single bit-test.
Also defer the span.kind tag read: previously read at the top of the publish
loop and threaded through both shouldComputeMetric and the inner publish.
isKind no longer needs the string, so the read can move down into the inner
publish where it's still needed for the SPAN_KINDS cache key / MetricKey.
Supporting changes:
- DDSpanContext.spanKindOrdinalOf(String) is now public so non-DDSpan CoreSpan
impls can compute the ordinal at tag-write time.
- SpanKindFilter gains a public matches(byte) fast-path overload that callers
with a pre-computed ordinal use directly.
- SimpleSpan caches the ordinal in setTag(SPAN_KIND, ...), mirroring what
TagInterceptor does for DDSpanContext, and its isKind now hits the byte
fast path. Without this, the JMH benchmark (which uses SimpleSpan) would
re-derive the ordinal on every isKind call and overstate the cost.
Benchmark on the bench updated last commit (kind=client on every span,
4 forks x 5 iter x 15s):
prior commit 6.585 ± 0.049 us/op
this commit 6.903 ± 0.096 us/op
The slight regression is a SimpleSpan-via-groovy-dispatch artifact -- the
interface call to isKind through CoreSpan, then through SimpleSpan, then
through SpanKindFilter.matches, doesn't fold as aggressively as a HashSet
contains on a static field. In production DDSpan.isKind inlines to a context
field read + ordinal byte read + bit-test, so the production path is faster
than the prior HashSet approach. A DDSpan-based benchmark would show this;
the existing SimpleSpan-based one doesn't.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add DDSpan-based variant of ConflatingMetricsAggregator JMH benchmark
The existing ConflatingMetricsAggregatorBenchmark uses SimpleSpan, a groovy
mock. That's enough for measuring queue/CHM/MetricKey work, but it conceals
the production cost of CoreSpan.isKind: SimpleSpan's isKind goes through
groovy interface dispatch into SpanKindFilter.matches, while DDSpan.isKind
inlines to a context byte-read + bit-test.
This new benchmark uses real DDSpan instances created through a CoreTracer
(with a NoopWriter so finishing doesn't reach the agent). Same shape as the
SimpleSpan bench (64-span trace, span.kind=client, peer.hostname set).
Numbers (2 forks x 5 iter x 15s):
master: 6.428 +- 0.189 us/op (HashSet eligibility checks)
this branch: 6.343 +- 0.115 us/op (SpanKindFilter bitmask)
About 1.3% faster on the production path. The SimpleSpan benchmark in the
same conditions shows a ~2.2% slowdown -- the mock's dispatch shape gives a
misleading signal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten SpanKindFilter encapsulation
Make SpanKindFilter.kindMask and its constructor private now that DDSpan.isKind
no longer needs direct field access -- it delegates to SpanKindFilter.matches(byte).
The Builder.build() in the same outer class still constructs instances via the
private constructor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Defer MetricKey construction and cache lookups to the aggregator thread
Replace the producer-side conflation pipeline with a thin per-span SpanSnapshot
posted to the existing aggregator thread. The aggregator now builds the
MetricKey, does the SERVICE_NAMES / SPAN_KINDS / PEER_TAGS_CACHE lookups, and
updates the AggregateMetric directly -- all off the producer's hot path.
What the producer does now, per span:
- filter (shouldComputeMetric, resource-ignored, longRunning)
- collect tag values into a SpanSnapshot (1 allocation per span)
- inbox.offer(snapshot) + return error flag for forceKeep
What moved off the producer:
- MetricKey construction and its hash computation
- SERVICE_NAMES.computeIfAbsent (UTF8 encoding of service name)
- SPAN_KINDS.computeIfAbsent (UTF8 encoding of span.kind)
- PEER_TAGS_CACHE lookups (peer-tag name+value UTF8 encoding)
- pending/keys ConcurrentHashMap operations
- Batch pooling, batch atomic ops, batch contributeTo
Removed entirely:
- Batch.java -- the conflation primitive is no longer needed; the
aggregator's existing LRUCache<MetricKey, AggregateMetric> IS the
conflation point now.
- pending ConcurrentHashMap<MetricKey, Batch>
- keys ConcurrentHashMap<MetricKey, MetricKey> (canonical dedup)
- batchPool MessagePassingQueue<Batch>
- The CommonKeyCleaner role of tracking keys.keySet() on LRU eviction --
AggregateExpiry now just reports drops to healthMetrics.
Added:
- SpanSnapshot: immutable value carrying the raw MetricKey inputs + a
tagAndDuration long (duration | ERROR_TAG | TOP_LEVEL_TAG).
- AggregateMetric.recordOneDuration(long tagAndDuration) -- the single-hit
equivalent of the existing recordDurations(int, AtomicLongArray).
- Peer-tag values flow through the snapshot as a flattened String[] of
[name0, value0, name1, value1, ...]; the aggregator encodes them through
PEER_TAGS_CACHE on its own thread.
Benchmark results (2 forks x 5 iter x 15s):
ConflatingMetricsAggregatorDDSpanBenchmark
prior commit 6.343 +- 0.115 us/op
this commit 2.506 +- 0.044 us/op (~60% faster)
ConflatingMetricsAggregatorBenchmark (SimpleSpan)
prior commit 6.585 +- 0.049 us/op
this commit 3.116 +- 0.032 us/op (~53% faster)
Caveat on the benchmark: without conflation, the producer pushes 1 inbox
item per span instead of ~1 per 64. At the benchmark's synthetic rate the
consumer can't keep up and inbox.offer silently drops. The numbers measure
producer publish() latency only; consumer throughput at realistic span rates
is a follow-up to validate. Tuning maxPending matters more in this design.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Report aggregator inbox-full drops via health metrics
With the per-span SpanSnapshot inbox path, the producer can lose snapshots
when the bounded MPSC queue is full -- silently, since inbox.offer() returns
a boolean we previously ignored. The conflating-Batch design used to absorb
~64x more producer pressure per inbox slot, so this is a new failure mode
worth surfacing.
Wire it through the existing HealthMetrics path:
- HealthMetrics.onStatsInboxFull() (no-op default).
- TracerHealthMetrics gets a statsInboxFull LongAdder and a new reason tag
reason:inbox_full reported under the same stats.dropped_aggregates metric
used for LRU evictions. Two LongAdders, two tagged time series.
- ConflatingMetricsAggregator.publish increments the counter when
inbox.offer(snapshot) returns false.
This doesn't fix the drop -- tuning maxPending and/or building producer-side
batching are the actual fixes. But it makes the failure visible in the same
place ops already watches.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'master' into dougqh/conflating-metrics-producer-wins
Merge branch 'dougqh/conflating-metrics-producer-wins' into dougqh/conflating-metrics-background-work
Add Hashtable and LongHashingUtils to datadog.trace.util
Two general-purpose utilities used by the client-side stats aggregator
work (PR #11382 and follow-ups), extracted into their own change so the
metrics-specific PRs can build on a smaller, reviewable foundation.
- Hashtable: a generic open-addressed-ish bucket table abstraction
keyed by a 64-bit hash, with a public abstract Entry type so client
code can subclass it for higher-arity keys. The metrics aggregator
uses it to back its AggregateTable.
- LongHashingUtils: chained 64-bit hash combiners with primitive
overloads (boolean, short, int, long, Object). Used in place of
varargs combiners to avoid Object[] allocation and boxing on the
hot path.
No callers within internal-api itself yet -- the metrics aggregator PR
will introduce the first usages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add AggregateTable + AggregateEntry backed by Hashtable
Standalone classes for swapping the consumer-side LRUCache<MetricKey,
AggregateMetric> with a multi-key Hashtable in the next commit. No call sites
use them yet.
- AggregateEntry extends Hashtable.Entry, holds the canonical MetricKey, the
mutable AggregateMetric, and copies of the 13 raw SpanSnapshot fields for
matches(). The 64-bit lookup hash is computed via chained
LongHashingUtils.addToHash calls (no varargs, no boxing of short/boolean).
- AggregateTable wraps a Hashtable.Entry[] from Hashtable.Support.create.
findOrInsert(SpanSnapshot) walks the bucket comparing raw fields, falling
back to MetricKeys.fromSnapshot on a true miss. On cap overrun, it scans
for an entry with hitCount==0 and unlinks it; if none, it returns null and
the caller drops the data point.
- MetricKeys.fromSnapshot extracts the canonicalization logic (DDCache
lookups + UTF8 encoding) from Aggregator.buildMetricKey, so the helper can
be called from AggregateTable on miss.
This also commits Hashtable and LongHashingUtils (added earlier, previously
uncommitted) and lifts Hashtable.Entry / Hashtable.Support visibility so
client code outside datadog.trace.util can build higher-arity tables -- the
case the javadoc describes but the original visibility didn't actually
support. Specifically: Entry is now public abstract with a protected ctor;
keyHash, next(), and setNext() are public; Support's create / clear /
bucketIndex / bucketIterator / mutatingBucketIterator methods are public.
Tests: AggregateTableTest covers hit, miss, distinct-by-spanKind, peer-tag
identity (including null vs non-null), cap overrun with stale victim, cap
overrun with no victim (returns null), expungeStaleAggregates, forEach,
clear, and that the canonical MetricKey is built at insert.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Swap LRUCache for AggregateTable in Aggregator + route disable() clear
Replace LRUCache<MetricKey, AggregateMetric> with the AggregateTable added
in the prior commit. The hot path in Drainer.accept becomes:
AggregateMetric aggregate = aggregates.findOrInsert(snapshot);
if (aggregate != null) {
aggregate.recordOneDuration(snapshot.tagAndDuration);
dirty = true;
} else {
healthMetrics.onStatsAggregateDropped();
}
On the steady-state hit path the lookup is a 64-bit hash compute + bucket
walk + matches(snapshot) -- no MetricKey allocation, no SERVICE_NAMES /
SPAN_KINDS / PEER_TAGS_CACHE lookups. The canonical MetricKey is now built
once per unique key at insert time, in MetricKeys.fromSnapshot.
Behavioral change in the cap-overrun path
-----------------------------------------
The old LRUCache evicted least-recently-used: at cap, a new insert would
push out the oldest entry regardless of whether it was live or stale.
AggregateTable instead scans for a hitCount==0 entry to recycle, and drops
the new key if none exists. Practical impact: in the common case where
the table holds a stable set of recurring keys, an unrelated burst of new
keys is dropped (and reported via onStatsAggregateDropped) rather than
evicting the established keys. The existing test that asserted "service0
evicted in favor of service10" is updated to assert the new semantics.
The other cap-related test ("should not report dropped aggregate when
evicted entry was already flushed") still passes unchanged: after report()
clears all entries to hitCount=0, the next wave of inserts recycles them.
Threading fix
-------------
ConflatingMetricsAggregator.disable() used to call aggregator.clearAggregates()
and inbox.clear() directly from the Sink's IO event thread, racing with the
aggregator thread mid-write. The race was tolerable for LinkedHashMap; it
is not for AggregateTable (chain corruption can NPE or loop). disable()
now offers a ClearSignal to the inbox so the aggregator thread itself
performs the table clear and the inbox.clear(). Adds one SignalItem
subclass + one branch in Drainer.accept; preserves the single-writer
invariant for AggregateTable end-to-end.
Removed: LRUCache import, AggregateExpiry inner class, the static
buildMetricKey / materializePeerTags / encodePeerTag helpers (now in
MetricKeys).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eliminate MetricKey: inline its fields onto AggregateEntry
MetricKey existed for two reasons -- the prior LRUCache key role (now handled
by AggregateTable's Hashtable.Entry mechanics) and as the labels argument
to MetricWriter.add. The first is gone; the second is the only thing keeping
MetricKey alive. Fold its UTF8-encoded label fields onto AggregateEntry,
change MetricWriter.add to take AggregateEntry directly, and delete
MetricKey + MetricKeys.
What AggregateEntry now holds
-----------------------------
- 10 UTF8BytesString label fields (resource, service, operationName,
serviceSource, type, spanKind, httpMethod, httpEndpoint, grpcStatusCode,
and a List<UTF8BytesString> peerTags for serialization).
- 3 primitives (httpStatusCode, synthetic, traceRoot).
- AggregateMetric (the value being accumulated).
- The raw String[] peerTagPairs is retained alongside the encoded peerTags
-- matches() compares it positionally against the snapshot's pairs; the
encoded form is only consumed by the writer.
matches(SpanSnapshot) compares the entry's UTF8 forms to the snapshot's raw
String / CharSequence fields via content-equality (UTF8BytesString.toString()
returns the underlying String in O(1)). This closes a latent bug in the
prior raw-vs-raw matches(): if one snapshot delivered a tag value as String
and a later snapshot delivered the same content as UTF8BytesString, the old
Objects.equals would return false and the table would split into two
entries. Content-equality matching collapses them into one.
Consolidated caches
-------------------
The static UTF8 caches that used to live partly on MetricKey (RESOURCE_CACHE,
OPERATION_CACHE, SERVICE_SOURCE_CACHE, TYPE_CACHE, KIND_CACHE,
HTTP_METHOD_CACHE, HTTP_ENDPOINT_CACHE, GRPC_STATUS_CODE_CACHE, SERVICE_CACHE)
and partly on ConflatingMetricsAggregator (SERVICE_NAMES, SPAN_KINDS,
PEER_TAGS_CACHE) are all now on AggregateEntry. The split was duplicating
work -- SERVICE_NAMES and SERVICE_CACHE both cached service-name to
UTF8BytesString. One cache per field now.
API change: MetricWriter.add
----------------------------
Was: add(MetricKey key, AggregateMetric aggregate)
Now: add(AggregateEntry entry)
The aggregate lives on the entry. Single-arg.
SerializingMetricWriter reads the same UTF8 fields off AggregateEntry that it
previously read off MetricKey; the wire format is byte-identical.
Test impact
-----------
AggregateEntry.of(...) takes the same 13 positional args new MetricKey(...)
took, so test diffs are mostly mechanical:
new MetricKey(args) -> AggregateEntry.of(args)
writer.add(key, _) -> writer.add(entry)
ValidatingSink in SerializingMetricWriterTest now iterates List<AggregateEntry>
directly. ConflatingMetricAggregatorTest's Spock matchers (~36 sites) rely
on AggregateEntry.equals comparing the 13 label fields (not the aggregate)
so the mock matches by labels regardless of the aggregate state at call time;
post-invocation closures verify aggregate state.
Benchmarks (2 forks x 5 iter x 15s)
-----------------------------------
The change is consumer-thread only; producer publish() is unchanged.
SimpleSpan bench: 3.123 +- 0.025 us/op (prior: 3.119 +- 0.018)
DDSpan bench: 2.412 +- 0.022 us/op (prior: 2.463 +- 0.041)
Both within noise -- the win is structural (one less class, one less
allocation per miss, one fewer cache layer) rather than benchmarked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add unit tests for Hashtable and LongHashingUtils
LongHashingUtilsTest (14 cases):
- hashCodeX null sentinel + non-null pass-through
- all primitive hash() overloads match the boxed Java hashCodes
- hash(Object...) 2/3/4/5-arg overloads match the chained addToHash
formula they are documented to constant-fold to
- addToHash(long, primitive) overloads match the Object-version
- linear-accumulation invariant (31 * h + v) holds across a sequence
- iterable / deprecated int[] / deprecated Object[] variants match
chained addToHash
- intHash treats null as 0 (observable via hash(null, "x"))
HashtableTest (24 cases across 5 nested classes):
- D1: insert/get/remove/insertOrReplace/clear/forEach, in-place value
mutation, null-key handling, hash-collision chaining with disambig-
uating equals, remove-from-collided-chain leaves siblings intact
- D2: pair-key identity, remove(pair), insertOrReplace matches on
both parts, forEach
- Support: capacity rounds up to a power of two, bucketIndex stays
in range across a wide hash sample, clear nulls every slot
- BucketIterator: walks only matching-hash entries in a chain, throws
NoSuchElementException when exhausted
- MutatingBucketIterator: remove from head-of-chain unlinks, replace
swaps the entry while preserving chain, remove() without prior
next() throws IllegalStateException
Tests live in internal-api/src/test/java/datadog/trace/util and use the
already-present JUnit 5 setup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply spotless formatting to Hashtable and LongHashingUtils
Bring the new util/ files in line with google-java-format
(tabs → spaces, line wrapping, javadoc list markup) so
spotlessCheck passes in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add JMH benchmarks for Hashtable.D1 and D2
Compares Hashtable.D1 and Hashtable.D2 against equivalent HashMap
usage for add, update, and iterate operations. Each benchmark thread
owns its own map (Scope.Thread), but @Threads(8) is used so the
allocation/GC pressure that Hashtable is designed to avoid surfaces
in the throughput numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add benchmark results to HashtableBenchmark header
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Guard Support.sizeFor against overflow and use Integer.highestOneBit;
reject capacities above 1 << 30 instead of looping forever.
- Add braces around single-statement while bodies in BucketIterator.
- Split HashtableBenchmark into HashtableD1Benchmark / HashtableD2Benchmark.
- Add regression tests for Support.sizeFor bounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix dropped argument in HashingUtils 5-arg Object hash
The 5-arg Object overload was forwarding only obj0..obj3 to the int
overload, silently dropping obj4. Also align LongHashingUtils.hash 3-arg
signature with its 2/4/5-arg siblings (int parameters) and strengthen
the 5-arg HashingUtilsTest to detect the missing-arg regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Split D1Tests and D2Tests into HashtableD1Test and HashtableD2Test;
extract shared test entry classes into HashtableTestEntries.
- Reduce visibility of LongHashingUtils.hash(int...) chaining overloads
to package-private; they are internal building blocks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop reflection in iterator tests via package-private D1.buckets
The iterator tests need a populated Hashtable.Entry[] to drive
Support.bucketIterator / mutatingBucketIterator. Relaxing D1.buckets
from private to package-private lets the same-package tests read it
directly, removing the reflection helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resize previousCounts for inbox-full health metric
The new reason:inbox_full reportIfChanged call advances countIndex to 51,
but previousCounts was still sized for 51 counters (max index 50), so the
metric never emitted and the resize warning fired every flush. Bump the
array to 52 and add a regression test that exercises the flush path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fold AggregateMetric into AggregateEntry
The label fields and the mutable counters/histograms are 1:1 with each
entry; carrying them on a separate object meant one extra allocation per
unique key plus an indirection on every hot-path update. Merging them
puts the counters directly on AggregateEntry, drops the entry.aggregate
hop, and consolidates ERROR_TAG / TOP_LEVEL_TAG onto the same class the
consumer uses to decode them.
AggregateTable.findOrInsert now returns AggregateEntry. Callers in
Aggregator and SerializingMetricWriter updated. Migrated
AggregateMetricTest.groovy to AggregateEntryTest.java per project policy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Avoid capturing lambda in Aggregator.report
Add a context-passing forEach(T, BiConsumer) overload to AggregateTable,
mirroring TagMap's pattern. Aggregator.report now hands the writer in as
context to a static BiConsumer so no fresh Consumer is allocated each
report cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add context-passing forEach to Hashtable.D1 and D2
Mirrors the TagMap pattern: pairs the existing forEach(Consumer) with a
forEach(T context, BiConsumer<T, TEntry>) overload so callers can hand
side-band state to a non-capturing lambda and avoid the
fresh-Consumer-per-call allocation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move forEach loop body to Support helper
Factors the unchecked (TEntry) cast out of D1.forEach / D2.forEach (and
the BiConsumer variants) into Support.forEach(buckets, ...). The cast
now lives in one place, mirroring how Entry.next() handles it, and the
D1/D2 methods become one-liners. Downstream higher-arity tables built
on Support gain the same helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Delegate AggregateTable.forEach to Support.forEach
Now that Hashtable.Support exposes the parameterized forEach helpers,
AggregateTable's own forEach methods can drop their duplicated loop body
and the (AggregateEntry) cast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move bucket-head cast to Support.bucket helper
Adds Support.bucket(buckets, keyHash) which returns the bucket head
already cast to the caller's concrete entry type. D1.get and D2.get
now drop the raw-Entry intermediate variable and walk the chain via
Entry.next() directly. The unchecked cast lives in one place,
consistent with Entry.next() and Support.forEach.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Use Support.bucket and type chain walks as AggregateEntry
- findOrInsert: walks via Support.bucket(buckets, keyHash) instead of
Hashtable.Entry + intermediate cast; bucketIndex is only computed on
the miss path now.
- evictOneStale / expungeStaleAggregates: chain variables typed as
AggregateEntry from the head down, leveraging Entry.next()'s generic
inference, so the per-iteration getHitCount() checks drop their
(AggregateEntry) cast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop d1_/d2_ prefix from per-table benchmark methods
Holdover from when both lived in a shared HashtableBenchmark; redundant
now that each lives in its own class.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add Hashtable.Support helpers: MAX_RATIO, insertHeadEntry, MutatingTableIterator
Three consumer-facing helpers that callers building higher-arity tables on
top of Hashtable.Support kept open-coding:
- MAX_RATIO_NUMERATOR / _DENOMINATOR: the 4/3 multiplier for sizing a
bucket array from a target working-set under a 75% load factor.
- insertHeadEntry(buckets, bucketIndex, entry): the (setNext + array-store)
pair for splicing a new entry at the head of a bucket chain.
- MutatingTableIterator + Support.mutatingTableIterator(buckets): walks
every entry in the table (not filtered by hash) with remove() support,
for sweeps like eviction and expunge that aren't keyed to a specific
hash. Sibling of MutatingBucketIterator.
Tests cover the table-wide iterator at head-of-bucket and mid-chain
removal, empty buckets between live entries, exhaustion, and
remove-without-next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Simplify AggregateTable via new Hashtable.Support helpers
- Constructor sizing now uses Support.MAX_RATIO_NUMERATOR / _DENOMINATOR
instead of an open-coded * 4 / 3.
- findOrInsert delegates the chain-head splice to Support.insertHeadEntry.
- evictOneStale and expungeStaleAggregates both rewritten in terms of
Support.mutatingTableIterator. Drops the bespoke head-vs-mid-chain
branching that read as more complicated than the operation actually is.
Net -28 lines in AggregateTable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Swap MAX_RATIO numerator/denominator pair for a single float + scaled create()
Replace Support.MAX_RATIO_NUMERATOR / _DENOMINATOR with a single float
MAX_RATIO constant, and add a Support.create(int, float) overload that
takes a scale factor. Callers now write Support.create(n, MAX_RATIO)
instead of stitching together the int arithmetic at the call site.
The scaled size is truncated (not ceiled) before going through sizeFor.
sizeFor already rounds up to the next power of two, so truncation just
absorbs float fuzz that would otherwise push a result like 12 * 4/3 =
16.0000005f past 16 and double the bucket array size for no reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Address second-round review on AggregateTable / Aggregator
- AggregateTable: switch to Support.create(maxAggregates, Support.MAX_RATIO)
now that the load-factor scaling is a Support concern.
- AggregateTable: replace open-coded "keyHash == X && matches(s)" with a
new AggregateEntry.matches(long keyHash, SpanSnapshot) overload that
bundles the hash gate.
- AggregateTable: rename local iterator var "it" -> "iter".
- Aggregator: drop WRITE_AND_CLEAR static field, inline as a non-capturing
lambda; the JIT reuses non-capturing lambdas, no need for the static
until a profile says otherwise.
- Aggregator: comment the ClearSignal branch with the thread-safety
rationale (single-writer invariant for AggregateTable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Hashtable docs + rename MAX_CAPACITY to MAX_BUCKETS
Five small cleanups from a design re-review pass:
1. Support javadoc: drop the stale "methods are package-private" sentence;
most of them were made public in earlier commits for higher-arity
callers. Also drop the "nested BucketIterator" framing (iterators are
peers of Support inside Hashtable, not nested inside Support).
2. MAX_RATIO javadoc: drop the Math.ceil recommendation; create(int, float)
deliberately truncates and is the canonical pathway.
3. Document the null-hash treatment on D1.Entry.hash and D2.Entry.hash so
the behavior difference is explicit: D1 uses Long.MIN_VALUE as a
sentinel that's collision-free against any int-valued hashCode(); D2
has no such sentinel and relies on matches() to resolve null/null vs
hash-0 collisions.
4. Rename Support.MAX_CAPACITY -> MAX_BUCKETS and sizeFor's parameter to
requestedSize. The cap is on the bucket-array length, not entry count;
the new name reflects that. Error messages updated to match.
5. Drop the `abstract` modifier on Hashtable in favor of `final` with a
private constructor. Nothing actually subclasses Hashtable -- the
abstract was a namespace device that read as "intended for extension."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dedupe chain-head splice in D1/D2 via keyHash insertHeadEntry overload
- Add Support.insertHeadEntry(buckets, long keyHash, entry) overload that
derives the bucket index itself. Callers that already have a hash but
not the index (the common case) now avoid the redundant bucketIndex(...)
hop.
- D1.insert, D1.insertOrReplace, D2.insert, D2.insertOrReplace: use the
new overload, drop the (thisBuckets local, bucketIndex compute,
setNext, store) sequence at each call site.
- D2.buckets: drop the `private` modifier to match D1.buckets. Both are
package-private so iterator tests in the same package can drive
Support.bucketIterator against the table's bucket array. Added a short
comment on both fields documenting the rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Entry.next encapsulation; doc hasNext; add D1/D2 getOrCreate
Three follow-ups from the design review:
- Make Hashtable.Entry.next private. All same-package readers
(BucketIterator) already had a next() accessor; the leftover direct
field reads now route through it. Closes the "mixed encapsulation"
gap where some readers used the accessor and same-package ones
reached for the field.
- BucketIterator and MutatingBucketIterator now document that chain-walk
work happens in next() (and the constructor for the first match);
hasNext() is an O(1) field read.
- Add D1.getOrCreate(K, Function) and D2.getOrCreate(K1, K2, BiFunction).
Both reuse the lookup hash for the insert on miss, avoiding the
double-hash that "get; if null then insert" callers would otherwise
pay.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Use keyHash insertHeadEntry overload in AggregateTable.findOrInsert
Picks up the Support.insertHeadEntry(buckets, long keyHash, entry)
overload added on the util-hashtable branch; saves the redundant
Support.bucketIndex(buckets, keyHash) hop at the call site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace // nullable comments with @Nullable annotations on AggregateEntry
Use javax.annotation.Nullable (the codebase's convention -- see DDSpan,
TagInterceptor, ScopeContext, etc.) on the four nullable label fields
(serviceSource, httpMethod, httpEndpoint, grpcStatusCode), their
getters, and the corresponding parameters of AggregateEntry.of.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop redundant load-factor comment from AggregateTable ctor
Support.MAX_RATIO and the scaled create(int, float) overload already
convey the 75% load-factor intent at the call site -- the inline
comment was duplicating their self-documentation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Import java.util.Objects in AggregateEntry instead of fully qualifying
Style nit -- the equals() method had eight fully-qualified references
to java.util.Objects.equals; add the import and drop the qualifier.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Document evictOneStale cost and disable() best-effort offer
Two design-review trade-offs that won't change in this PR but should be
explicit at the call sites:
- AggregateTable.evictOneStale: O(N) scan per call (vs LRUCache's O(1)),
acceptable because the new policy drops the *new* key on cap-overrun
rather than evicting an established one -- so eviction is expected to
be rare. Cursor-caching is the future optimization if a workload runs
persistently at cap.
- ConflatingMetricsAggregator.disable: single inbox.offer(CLEAR) is
best-effort. If the inbox is full the clear is dropped, but the
system self-heals (supportsMetrics() is already false, the next
report-sink-rejection retries disable). Worst case is one extra cycle
of stale data, not a leak.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skip SpanSnapshot allocation when the inbox is already at capacity
publish() previously did all of the tag extraction (peer-tag pairs,
HTTP method/endpoint, span kind, gRPC status) and the SpanSnapshot
allocation before calling inbox.offer; on a full inbox the offer
failed and everything became garbage.
Early-out with an approximate size() vs capacity() check up front. The
jctools MPSC queue's size() is best-effort but that's fine: under-
estimation falls through to the existing offer-as-source-of-truth
path, over-estimation drops a snapshot that would have fit (and
onStatsInboxFull was about to fire on the next span anyway).
error is computed first so the force-keep return is correct whether
or not the snapshot is built.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hashtable: add missing braces and detach removed/replaced entries
Addresses PR #11409 review comments:
- #3267164119 / #3267165525: wrap every single-line if/break body in
braces (7 sites across BucketIterator, MutatingBucketIterator, and the
full-table Iterator).
- #3275947761 / #3275948108 (sarahchen6): null out the removed/replaced
entry's next pointer after splicing it out of the chain in
MutatingBucketIterator.remove / .replace. Applied the same fix to the
full-table Iterator.remove for consistency.
Rationale: detaching prevents accidental traversal through a removed
entry via a stale reference and lets the GC reclaim a chain tail that
the removed entry was the last referrer to.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Add Hashtable and LongHashingUtils to datadog.trace.util
Two general-purpose utilities used by the client-side stats aggregator
work (PR #11382 and follow-ups), extracted into their own change so the
metrics-specific PRs can build on a smaller, reviewable foundation.
- Hashtable: a generic open-addressed-ish bucket table abstraction
keyed by a 64-bit hash, with a public abstract Entry type so client
code can subclass it for higher-arity keys. The metrics aggregator
uses it to back its AggregateTable.
- LongHashingUtils: chained 64-bit hash combiners with primitive
overloads (boolean, short, int, long, Object). Used in place of
varargs combiners to avoid Object[] allocation and boxing on the
hot path.
No callers within internal-api itself yet -- the metrics aggregator PR
will introduce the first usages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add unit tests for Hashtable and LongHashingUtils
LongHashingUtilsTest (14 cases):
- hashCodeX null sentinel + non-null pass-through
- all primitive hash() overloads match the boxed Java hashCodes
- hash(Object...) 2/3/4/5-arg overloads match the chained addToHash
formula they are documented to constant-fold to
- addToHash(long, primitive) overloads match the Object-version
- linear-accumulation invariant (31 * h + v) holds across a sequence
- iterable / deprecated int[] / deprecated Object[] variants match
chained addToHash
- intHash treats null as 0 (observable via hash(null, "x"))
HashtableTest (24 cases across 5 nested classes):
- D1: insert/get/remove/insertOrReplace/clear/forEach, in-place value
mutation, null-key handling, hash-collision chaining with disambig-
uating equals, remove-from-collided-chain leaves siblings intact
- D2: pair-key identity, remove(pair), insertOrReplace matches on
both parts, forEach
- Support: capacity rounds up to a power of two, bucketIndex stays
in range across a wide hash sample, clear nulls every slot
- BucketIterator: walks only matching-hash entries in a chain, throws
NoSuchElementException when exhausted
- MutatingBucketIterator: remove from head-of-chain unlinks, replace
swaps the entry while preserving chain, remove() without prior
next() throws IllegalStateException
Tests live in internal-api/src/test/java/datadog/trace/util and use the
already-present JUnit 5 setup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply spotless formatting to Hashtable and LongHashingUtils
Bring the new util/ files in line with google-java-format
(tabs → spaces, line wrapping, javadoc list markup) so
spotlessCheck passes in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add JMH benchmarks for Hashtable.D1 and D2
Compares Hashtable.D1 and Hashtable.D2 against equivalent HashMap
usage for add, update, and iterate operations. Each benchmark thread
owns its own map (Scope.Thread), but @Threads(8) is used so the
allocation/GC pressure that Hashtable is designed to avoid surfaces
in the throughput numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add benchmark results to HashtableBenchmark header
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Guard Support.sizeFor against overflow and use Integer.highestOneBit;
reject capacities above 1 << 30 instead of looping forever.
- Add braces around single-statement while bodies in BucketIterator.
- Split HashtableBenchmark into HashtableD1Benchmark / HashtableD2Benchmark.
- Add regression tests for Support.sizeFor bounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix dropped argument in HashingUtils 5-arg Object hash
The 5-arg Object overload was forwarding only obj0..obj3 to the int
overload, silently dropping obj4. Also align LongHashingUtils.hash 3-arg
signature with its 2/4/5-arg siblings (int parameters) and strengthen
the 5-arg HashingUtilsTest to detect the missing-arg regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Split D1Tests and D2Tests into HashtableD1Test and HashtableD2Test;
extract shared test entry classes into HashtableTestEntries.
- Reduce visibility of LongHashingUtils.hash(int...) chaining overloads
to package-private; they are internal building blocks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop reflection in iterator tests via package-private D1.buckets
The iterator tests need a populated Hashtable.Entry[] to drive
Support.bucketIterator / mutatingBucketIterator. Relaxing D1.buckets
from private to package-private lets the same-package tests read it
directly, removing the reflection helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add context-passing forEach to Hashtable.D1 and D2
Mirrors the TagMap pattern: pairs the existing forEach(Consumer) with a
forEach(T context, BiConsumer<T, TEntry>) overload so callers can hand
side-band state to a non-capturing lambda and avoid the
fresh-Consumer-per-call allocation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move forEach loop body to Support helper
Factors the unchecked (TEntry) cast out of D1.forEach / D2.forEach (and
the BiConsumer variants) into Support.forEach(buckets, ...). The cast
now lives in one place, mirroring how Entry.next() handles it, and the
D1/D2 methods become one-liners. Downstream higher-arity tables built
on Support gain the same helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move bucket-head cast to Support.bucket helper
Adds Support.bucket(buckets, keyHash) which returns the bucket head
already cast to the caller's concrete entry type. D1.get and D2.get
now drop the raw-Entry intermediate variable and walk the chain via
Entry.next() directly. The unchecked cast lives in one place,
consistent with Entry.next() and Support.forEach.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop d1_/d2_ prefix from per-table benchmark methods
Holdover from when both lived in a shared HashtableBenchmark; redundant
now that each lives in its own class.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add Hashtable.Support helpers: MAX_RATIO, insertHeadEntry, MutatingTableIterator
Three consumer-facing helpers that callers building higher-arity tables on
top of Hashtable.Support kept open-coding:
- MAX_RATIO_NUMERATOR / _DENOMINATOR: the 4/3 multiplier for sizing a
bucket array from a target working-set under a 75% load factor.
- insertHeadEntry(buckets, bucketIndex, entry): the (setNext + array-store)
pair for splicing a new entry at the head of a bucket chain.
- MutatingTableIterator + Support.mutatingTableIterator(buckets): walks
every entry in the table (not filtered by hash) with remove() support,
for sweeps like eviction and expunge that aren't keyed to a specific
hash. Sibling of MutatingBucketIterator.
Tests cover the table-wide iterator at head-of-bucket and mid-chain
removal, empty buckets between live entries, exhaustion, and
remove-without-next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Swap MAX_RATIO numerator/denominator pair for a single float + scaled create()
Replace Support.MAX_RATIO_NUMERATOR / _DENOMINATOR with a single float
MAX_RATIO constant, and add a Support.create(int, float) overload that
takes a scale factor. Callers now write Support.create(n, MAX_RATIO)
instead of stitching together the int arithmetic at the call site.
The scaled size is truncated (not ceiled) before going through sizeFor.
sizeFor already rounds up to the next power of two, so truncation just
absorbs float fuzz that would otherwise push a result like 12 * 4/3 =
16.0000005f past 16 and double the bucket array size for no reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Hashtable docs + rename MAX_CAPACITY to MAX_BUCKETS
Five small cleanups from a design re-review pass:
1. Support javadoc: drop the stale "methods are package-private" sentence;
most of them were made public in earlier commits for higher-arity
callers. Also drop the "nested BucketIterator" framing (iterators are
peers of Support inside Hashtable, not nested inside Support).
2. MAX_RATIO javadoc: drop the Math.ceil recommendation; create(int, float)
deliberately truncates and is the canonical pathway.
3. Document the null-hash treatment on D1.Entry.hash and D2.Entry.hash so
the behavior difference is explicit: D1 uses Long.MIN_VALUE as a
sentinel that's collision-free against any int-valued hashCode(); D2
has no such sentinel and relies on matches() to resolve null/null vs
hash-0 collisions.
4. Rename Support.MAX_CAPACITY -> MAX_BUCKETS and sizeFor's parameter to
requestedSize. The cap is on the bucket-array length, not entry count;
the new name reflects that. Error messages updated to match.
5. Drop the `abstract` modifier on Hashtable in favor of `final` with a
private constructor. Nothing actually subclasses Hashtable -- the
abstract was a namespace device that read as "intended for extension."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dedupe chain-head splice in D1/D2 via keyHash insertHeadEntry overload
- Add Support.insertHeadEntry(buckets, long keyHash, entry) overload that
derives the bucket index itself. Callers that already have a hash but
not the index (the common case) now avoid the redundant bucketIndex(...)
hop.
- D1.insert, D1.insertOrReplace, D2.insert, D2.insertOrReplace: use the
new overload, drop the (thisBuckets local, bucketIndex compute,
setNext, store) sequence at each call site.
- D2.buckets: drop the `private` modifier to match D1.buckets. Both are
package-private so iterator tests in the same package can drive
Support.bucketIterator against the table's bucket array. Added a short
comment on both fields documenting the rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Entry.next encapsulation; doc hasNext; add D1/D2 getOrCreate
Three follow-ups from the design review:
- Make Hashtable.Entry.next private. All same-package readers
(BucketIterator) already had a next() accessor; the leftover direct
field reads now route through it. Closes the "mixed encapsulation"
gap where some readers used the accessor and same-package ones
reached for the field.
- BucketIterator and MutatingBucketIterator now document that chain-walk
work happens in next() (and the constructor for the first match);
hasNext() is an O(1) field read.
- Add D1.getOrCreate(K, Function) and D2.getOrCreate(K1, K2, BiFunction).
Both reuse the lookup hash for the insert on miss, avoiding the
double-hash that "get; if null then insert" callers would otherwise
pay.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hashtable: add missing braces and detach removed/replaced entries
Addresses PR #11409 review comments:
- #3267164119 / #3267165525: wrap every single-line if/break body in
braces (7 sites across BucketIterator, MutatingBucketIterator, and the
full-table Iterator).
- #3275947761 / #3275948108 (sarahchen6): null out the removed/replaced
entry's next pointer after splicing it out of the chain in
MutatingBucketIterator.remove / .replace. Applied the same fix to the
full-table Iterator.remove for consistency.
Rationale: detaching prevents accidental traversal through a removed
entry via a stale reference and lets the GC reclaim a chain tail that
the removed entry was the last referrer to.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename LongHashingUtils.hashCodeX(Object) to hash(Object) for API consistency
Addresses PR #11409 review comment #3276167001. The method parallels the
primitive hash(boolean) / hash(int) / hash(long) / ... family, so naming
it hash(Object) -- with null collapsing to Long.MIN_VALUE as a sentinel
distinct from any real hashCode -- matches the rest of the public surface.
Test call sites that pass a literal null now disambiguate against
hash(int[]) / hash(Object[]) / hash(Iterable) via an (Object) cast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge remote-tracking branch 'origin/master' into dougqh/optimize-metric-key
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Merge remote-tracking branch 'origin/master' into dougqh/conflating-metrics-background-work
Introduce slim PeerTagSchema; capture peer-tag values not pairs
Addresses sarahchen6's review comment on ConflatingMetricsAggregator
extractPeerTagPairs: replaces the worst-case-allocation + trim-and-copy
flat-pairs layout with a parallel-array carrier.
- New PeerTagSchema: minimal carrier of String[] names. Two flavors -- a
static INTERNAL singleton (one entry: base.service) for internal-kind
spans, and per-discovery built schemas for client/producer/consumer
spans. Deliberately no cardinality limiters or per-cycle state; that
layers on top in a later PR.
- ConflatingMetricsAggregator: caches the peer-aggregation schema keyed
on reference equality of features.peerTags() -- a single volatile read
+ a long compare on the steady-state producer hot path, no allocation.
The producer now captures only a String[] of values parallel to the
schema's names; the schema reference is carried on SpanSnapshot. The
prior "build worst-case pairs then trim" code is gone.
- SpanSnapshot: replaces String[] peerTagPairs with PeerTagSchema +
String[] peerTagValues. Producer drops the schema reference if no
values fired so the consumer short-circuits on null.
- Aggregator.materializePeerTags: now reads name/value pairs at the same
index from (schema.names, snapshot.peerTagValues). Counts hits once
for exact-size allocation; preserves the singletonList fast path for
the common one-entry case (e.g. internal-kind base.service).
Producer-side cost goes from "allocate String[2n] + walk + maybe trim"
to "single volatile read + walk + lazy String[n] only on first hit".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Address PR #11381 review (round 2)
- Aggregator.materializePeerTags: fold the firstHit-discovery nested if
into a single guarded post-increment (amarziali, #3279243138). One
body line: `if (values[i] != null && hitCount++ == 0) firstHit = i;`.
- Drop redundant isKind(SpanKindFilter) overrides in both
TraceGenerator.groovy files (amarziali, #3279264553 / #3279382648).
CoreSpan.java:84 already supplies a default implementation that reads
the same span.kind tag.
- Bump TRACER_METRICS_MAX_PENDING default from 2048 -> 131072 to address
the capacity regression amarziali flagged (#3279378375). Without
producer-side conflation, the inbox now holds 1 SpanSnapshot per
metrics-eligible span instead of 1 conflated Batch per ~64 spans;
restoring effective capacity parity (~2048 * ~64 = 131072) prevents a
~64x rise in inbox-full drops at the same span rate. ~100 B per
SpanSnapshot puts the worst-case heap floor at ~13 MB -- bounded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cover inbox-full fast-path in ConflatingMetricsAggregator.publish
Addresses PR #11381 review (amarziali, #3279325340 -- "Are the existing
tests covering this case?").
New ConflatingMetricsAggregatorInboxFullTest constructs the aggregator
with a small inbox (queueSize=8), deliberately does NOT call start() so
the consumer thread never drains, then publishes enough spans to
overflow the inbox. Verifies that healthMetrics.onStatsInboxFull() is
called at least once -- the fast-path's `inbox.size() >= inbox.capacity()`
short-circuit triggers when the producer-side queue is at capacity.
Test is Java + JUnit 5 + Mockito per the project convention for new
tests; uses a CoreSpan Mockito mock rather than the SimpleSpan Groovy
fixture so we don't depend on Groovy-then-Java compile order from the
test source set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reconcile PeerTagSchema once per reporting cycle on the aggregator thread
Addresses amarziali's review comment #3279340181 ("It would be more
efficient to trigger from the other side"). The producer-side reference
compare on every publish goes away; the aggregator thread reconciles
the cached schema against feature discovery once per reporting cycle.
- DDAgentFeaturesDiscovery: expose getLastTimeDiscovered() so callers
can detect a discovery refresh without copying the peerTags Set.
- PeerTagSchema: add `long lastTimeDiscovered` (plain, aggregator-only)
and `hasSameTagsAs(Set)`. of(Set, long) takes the timestamp; INTERNAL
uses a -1L sentinel since it's never reconciled.
- ConflatingMetricsAggregator:
* Drop the cachedPeerTagsSource volatile and the per-publish reference
compare.
* Producer fast path is now `cachedPeerTagSchema` volatile read +
null-check; first publish takes the one-time synchronized bootstrap.
* Add reconcilePeerTagSchema() that runs once per cycle on the
aggregator thread: fast-path timestamp compare, slow-path set
compare, bump-in-place when the set is unchanged.
- Aggregator: new `Runnable onReportCycle` constructor parameter, run at
the start of report() (before the flush, so any test awaiting
writer.finishBucket() observes the schema in its post-reconcile state
and so the next publish sees the new schema without a handoff).
- Update "should create bucket for each set of peer tags" to drive two
reporting cycles separated by a report() that triggers reconcile. The
old test relied on per-publish reference detection, which the new
design intentionally doesn't preserve -- the schema is now stable
within a cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Add bootstrap + reconcile coverage for PeerTagSchema
Addresses round-3 review nice-to-haves on PR #11381.
- PeerTagSchemaTest: unit coverage for hasSameTagsAs() (the predicate
that drives the reconcile fast/slow path split), the of(Set, long)
factory, and the INTERNAL singleton. The hasSameTagsAs cases include
same-content-different-Set-reference (the case the reconcile fast path
relies on after a discovery refresh) and content-mismatch in either
direction.
- ConflatingMetricsAggregatorBootstrapTest: integration coverage for
the producer-side bootstrap + aggregator-thread reconcile flow.
* bootstrapHappensOnceOnFirstPublish -- three publishes against an
un-started aggregator (no consumer thread, no reconciles); verifies
features.peerTags() and features.getLastTimeDiscovered() are each
called exactly once.
* reconcileSkipsDeepCompareWhenTimestampMatches -- two cycles with
constant features.getLastTimeDiscovered(); each post-report
reconcile short-circuits on the timestamp fast path, so peerTags()
is called only by bootstrap (1 total).
* reconcileSurvivesTimestampBumpWhenTagsUnchanged -- timestamps bump
every reconcile, forcing the slow set-compare path; the tag set
stays identical, so the schema is preserved and continues to flush
buckets correctly across cycles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Use writer.finishBucket() count in bootstrap test for cascade compatibility
The verify(writer).add(MetricKey, AggregateMetric) signature is unique
to #11381; downstream branches use AggregateEntry. Switching to
verify(writer, times(2)).finishBucket() keeps the same behavioral
guarantee (both cycles flushed) across the stack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Use writer.finishBucket() count in bootstrap test for cascade compatibility
The verify(writer).add(MetricKey, AggregateMetric) signature is unique
to #11381; downstream branches use AggregateEntry. Switching to
verify(writer, times(2)).finishBucket() keeps the same behavioral
guarantee (both cycles flushed) across the stack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'master' into dougqh/conflating-metrics-background-work
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Preserve TRACER_METRICS_MAX_PENDING semantic + drop stale imports
TRACER_METRICS_MAX_PENDING previously counted conflating Batch slots
(~64 spans each). The inbox now holds 1 SpanSnapshot per slot, so
multiply the configured value by LEGACY_BATCH_SIZE (64) to keep
pre-existing customer overrides delivering the same effective
span-throughput capacity. Default stays at 2048 logical -> 131072
snapshot slots, identical to the prior 2048 batches * 64 spans.
Also drops two unused datadog.trace.core.SpanKindFilter imports left
behind in TraceGenerator.groovy after the isKind() override was removed
in favor of the CoreSpan default implementation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Add AdversarialMetricsBenchmark for capacity-bound stress testing
Ports the adversarial JMH benchmark from #11402 down to this branch so
we can compare #11381 vs master on a high-cardinality, high-throughput
workload. Adapted to use ConflatingMetricsAggregator (pre-rename) and
the FixedAgentFeaturesDiscovery / NullSink helpers already in
ConflatingMetricsAggregatorBenchmark.
8 producer threads hammer publish() with unique (service, operation,
resource, peer.hostname) per op so the aggregate cache fills+evicts
continuously and the inbox saturates. tearDown prints the drop
counters (inboxFull vs aggregateDropped) so the test verifies the
subsystem stayed bounded under attack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Trim AdversarialMetricsBenchmark counters and clarify printout
Drop traceComputedCalls / totalSpansCounted: under 8-way contention
the volatile-long ++/+= pattern was losing ~20% of updates (296M
counted vs 245M reported), and the numbers duplicate signal JMH's
ops/s already provides.
Switch inboxFull / aggregateDropped to LongAdder so the printed drop
shape (the order-of-magnitude story the bench is built to tell) is
accurate under contention.
Replace the stale "both forks combined for this run" string with text
that matches the actual @Fork(value=1) config and notes that counters
accumulate across warmup + measurement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Close PeerTagSchema reconcile race + cover the swap branch
buildPeerTagSchema previously read features.peerTags() before
features.getLastTimeDiscovered(). DDAgentFeaturesDiscovery exposes
those as two separate accessors against its volatile State -- a
state-swap interleaving could leave the cached schema tagged with a
NEWER timestamp than its names, after which the next reconcile
short-circuits on the timestamp compare and misses the tag-set update
until the next discovery refresh (~minute later).
Swap the read order so timestamp is captured first. With this
ordering, an interleaving leaves the schema OLDER than its names
instead -- the next reconcile sees a timestamp mismatch, runs the
deep compare, and self-heals on the very next cycle.
Also adds reconcileSwapsSchemaWhenTagSetChanges, which closes the
test gap on the slow-path swap branch
(cachedPeerTagSchema = PeerTagSchema.of(...)). End-to-end check via
the writer's captured MetricKeys: pre-swap snapshot carries only
peer.hostname, post-swap snapshot carries both peer.hostname and
peer.service.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Adapt reconcileSwapsSchemaWhenTagSetChanges to AggregateEntry shape
#11382 collapses MetricWriter.add(MetricKey, AggregateMetric) into
add(AggregateEntry). Re-target the captor and accessors on this branch
so the test compiles and the same end-to-end peer-tag verification
holds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Clarify materializePeerTags hit-counting loop
Splits the `if (values[i] != null && hitCount++ == 0)` conjunction
into nested ifs. Same semantics, no codegen impact after JIT --
just visibly says what the loop is doing rather than relying on
post-increment-inside-conjunction. Closes amarziali's review thread
on this block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
# Conflicts:
# dd-trace-core/src/main/java/datadog/trace/common/metrics/Aggregator.java
Fix MetricsIntegrationTest entry recording call site
AggregateEntry consolidated MetricKey + AggregateMetric so recordDurations
lives directly on AggregateEntry now. The previous entry1.aggregate.
recordDurations(...) form compiles under Groovy's dynamic dispatch but
would throw MissingPropertyException at runtime since there is no
`aggregate` property. Resolves chatgpt-codex-connector's review comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make ConflatingMetricAggregatorTest counter checks actually verify
The `1 * writer.add(value) >> { closure }` pattern treats the closure
as a stubbed return value -- Spock evaluates it but discards the
result, so `e.getHitCount() == X && ...` was a silent no-op across
31 occurrences. Wrapping the expression in `assert` makes Groovy's
power-assert throw on mismatch, which Spock surfaces as a real
failure. Resolves chatgpt-codex-connector's review comment.
All 41 tests still pass, so the previously-unverified assertions
happened to hold.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop dead recordDurations(int, AtomicLongArray) batch API
This method was a vestige of master's Batch design where multiple
producer threads wrote into an AtomicLongArray slot concurrently and
the aggregator drained ~64 durations per Batch in one call. The new
producer/consumer split publishes one SpanSnapshot per span, so
production only ever calls recordOneDuration(long).
Migrate the three remaining callers (AggregateEntryTest,
SerializingMetricWriterTest, MetricsIntegrationTest) to a loop of
recordOneDuration(long) calls, then delete the batched method and its
AtomicLongArray imports.
Drops the recordDurationsIgnoresTrailingZeros test -- that behavior
was a specific quirk of the batched API (count parameter shorter than
the array length) and doesn't apply to recordOneDuration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Warn about colon split in AggregateEntry.of test factory
The factory recovers (name, value) pairs from pre-encoded "name:value"
strings by splitting at the FIRST colon. Test-only, but worth being
explicit so callers don't hand it a peer-tag value containing a colon
(URLs, IPv6, service:env) and get a silently wrong (name, value) pair.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add coverage for disable() -> ClearSignal threading path
The bundled fix in this PR routes the agent-downgrade clear through
the inbox so the aggregator thread stays the sole writer to
AggregateTable. Prior to this test, there was no regression coverage
for that routing.
The test fires DOWNGRADED from the test thread (production-like
OkHttpSink callback path), waits for the immediate no-flush window,
then publishes a marker span with a distinct resource name. The
subsequent report's writer.add captor must see only the marker -- if
CLEAR didn't actually wipe the original entry, the original
"resource" would still be present and the assertion would catch it.
Cannot directly verify thread identity of the clear from inside this
test (CLEAR's inbox.clear() drops any latch signal we'd queue behind
it), so this is an observable-contract test rather than a strict
thread-id test. Still catches both the missing-clear regression and
the …
What Does This Do
Introduces Hashtable which serves as lighter weight alternative to HashMap.
Motivation
Hashtable is parameterized on Entry types allowing for lower overhead.
The Entry can hold multiple fields that comprise the key.
The Entry can hold mutable fields that compromise the value.
The Entry can include metainfo useful for eviction, etc
Hashable includes D1 and D2 for 1-D and 2-D maps respectively, but also includes a Support class that be used to make higher dimensional / more complicated map structures.
Particularly useful in aggregation workloads with multipart keys where lookups dominate insertions. In those situations, a solution based on Hashtable avoids constantly allocating a composite key object that will be immediately thrown away.
Additional Notes
Splits out of #11382 into stand-alone own change:
datadog.trace.util.Hashtable— generic open-addressed-ish bucket table keyed by a 64-bit hash. Public abstractEntrylets client code subclass it for higher-arity keys (e.g. for multi-field aggregation keys in the metrics aggregator). Support helpers (create,clear,bucketIndex,bucketIterator,mutatingBucketIterator) are package-private but enough for higher layers built on top.datadog.trace.util.LongHashingUtils— chained 64-bit hash combiners with primitive overloads (boolean,short,int,long,Object). Used in place of varargs combiners to avoidObject[]allocation and boxing on the hot path.No callers within
internal-apiyet. The first usage will land in #11382 (AggregateTable+AggregateEntry), which now becomes a smaller, more focused diff once this lands first.Test plan
:internal-api:compileJavapasses (verified locally).🤖 Generated with Claude Code